Pipeline, July/August 1995, vol.6, no.4
Copyright © 1995 Silicon Graphics
This article will present a general discussion on tuning IRIX kernels as well as the preparation for, and recovery from, an unbootable kernel. It is not an exhaustive study of the subject, and system administrators are encouraged to read the references listed at the end of this article.
This article assumes that the reader is an experienced system administrator on SGI platforms. Although many of the concepts and concerns presented in this article are common to all Unix systems, the procedures are specific to IRIX.
The reader should be familiar with chapter 11, "System Performance Tuning" in the IRIX Admin: System Configuration and Operation Manual, as well as the IRIX Admin: Software Installation and Licensing Manual (both of which are available on-line with InSight(1)).
Note that although some references will be made to IRIX 4.0.x, this article is primarily concerned with IRIX 5.x and IRIX 6.0.x. In addition, all examples in this article will assume SCSI devices.
Finally, modifying a kernel under IRIX should not be done lightly. Care is taken by Silicon Graphics to ensure that the kernel is well tuned for most environments when delivered from the factory.
Preparing for Recovery
Before any modifications are made to the kernel, the system administrator should prepare and test a procedure to recover from an unbootable kernel, or to return to a prior configuration. Failure to do so may result in extended system downtime, and a call to your support provider.
At a minimum, the system administrator should have a SGI CD-ROM drive and the correct IRIX CD-ROM for the currently running operating system. Alternately, a bootable software distribution directory containing the correct version of IRIX on the local network can be used. If neither of these are available, the system administrator should delay reconfiguring the IRIX kernel.
Save the Current Kernel Settings
A copy of the last known working kernel parameters should be kept either on another system or in hardcopy. This will allow the system administrator to rebuild a correctly configured kernel if necessary. Over time, this information may also help the system administrator understand usage trends on the system.
The systune(1M) command can be used to print the majority of the commonly modified kernel parameters. The exceptions are some of the networking and SCSI parameters defined in the files in the directory /var/sysgen/master.d. These files should be backed up on tape, printed out, or copied to another system for safety.
The following command will save a copy of the output from systune in the root directory and send a copy of the output to the system's default printer:
# systune | pr -h "systune output" | \ tee /systune.before | lp
Save a Copy of a Working Kernel
The possibility exists that modifying the kernel parameters may create a kernel that cannot be booted, or that runs extremely poorly. There are several ways to recover from this predicament. The ease of the recovery depends on how well the system administrator has prepared for this situation.
If there is space in the root filesystem, a copy of the current running kernel can be saved and booted from should the newly tuned kernel fail to boot or to run well. The section "Changing Parameters and Reconfiguring the System" in Chapter 11, "System Performance Tuning" of the IRIX Admin: System Configuration and Operation Manual provides details on how to do this.
If the system does not have sufficient free space in the root filesystem, it is possible to make a copy of the kernel on an alternate filesystem. For example, if the system has a large amount of free space on the option drive, a copy of the kernel can be saved there. In this case copy the kernel to /disk2:
# df Filesystem Type blocks use avail %use Mounted on /dev/root efs 1976624 1091936 884688 55% / /dev/dsk/dks0d2s7 efs 2058160 700024 1358136 34% /disk2 # cp /unix /disk2/unix.saveThe system administrator should make sure that the alternate drive can be seen by the hinv command in the Command Monitor. If a drive is not listed (such as a drive on a jaguar (VME-SCSI) controller), filesystems on that drive should not be used to save the kernel. Save the kernel in the root directory of the filesystem, and not a subdirectory (e.g., don't use /disk2/stuff/unix.save). Saving the kernel on a logical volume is not recommended since these filesystems may not be accessible in the Command Monitor.
If desired, the system administrator can also write the backup kernel to local tape using tar(1), bru(1), or cpio(1). Be sure to read the tape after making it to ensure that the backup is indeed good. Note that IRIS NetWorker archives are not accessible in miniroot and should not be used in this procedure.
To ensure minimal downtime, the system administrator should prepare and test a procedure to recover from an unbootable kernel, or to return to a prior configuration before any modifications are made to the kernel.
Kernel Tuning: Tunable Parameters
The following section will describe some of the most commonly modified kernel parameters. The configuration method for some kernel parameters has changed between IRIX 4.0.x and IRIX 5.x (and IRIX 6.0.x). For those parameters, a brief description of the changes is also provided. Also note that almost all kernel parameters are now specified in lower case.
Additional information on these and other parameters is provided in the IRIX Admin: System Configuration and Operation Manual.
Kernel Switches
There are some tunable parameters that are simply switches. The switch can be on or off.
availsmem_accountingIRIX 4.0.x provided a switch called availsmem_accounting to turn on or off strict swap space accounting. By default this switch was turned off which meant that programs could request more memory than was actually available on the system. In general, this worked because most of the programs did not use all the memory that they requested. If the system ran out of swap space the system would make the space available by terminating processes with a low priority.
In IRIX 5.x and IRIX 6.0.x, the availsmem_accounting switch has been removed and replaced by a virtual swap option, called vswap (refer to the manual page for swap(1M) for more information). This is not a tunable switch, but is an option that can be turned on or off through chkconfig(1M). Under IRIX 5.2, this switch is off on all systems except the Indy and Indigo2. Under IRIX 5.3, this switch is off for all systems. Use the following command to check the state of the flag:
# /etc/chkconfig | grep vswap
vswap on
If it is on, as it is in this example, the kernel will allow a process to load and run that could in theory take up to (by default) 40MB (80,000 512 byte blocks) of additional memory that the system does not have. The amount of virtual swap that can be allocated is configurable by modifying the variable vswaplen in the file /var/config/vswap.options. For more information on the use of virtual swap, refer to the manual page for swap(1M) and chkconfig(1M), and the section called "Swap Space in chapter 6 ("Configuring Disk and Swap Space") of the IRIX Admin: System Configuration and Operation Manual.
nosuidshells
IRIX 5.x and IRIX 6.0.x continue to provide the switch nosuidshells which controls whether shell scripts with the set-user-id bit set to root are allowed to run. By default, this switch is set to 1 (off) because of the security risks associated with setuid scripts and programs.
restricted_chown
IRIX 5.x and IRIX 6.0.x continue to provide the restricted_chown switch which controls whether users can use chown(2) to change the ownership of a file to another user. By default, this switch is set to 0 (on) which means that any user can change the ownership of a file that they own to another user. This switch is most often changed on systems that have disk quotas(4) turned on and do not want users to avoid the disk quota system by changing file ownership to another user.
For more information on disk quotas, refer to the article titled "Controlling Disk Resources: Quotas" in the January/February 1994 issue of Pipeline (available on-line with InSight if Support Advantage is loaded).
Streams Parameters
Under IRIX 4.0.x, the parameters NBLK4 through NBLK4096 were used to control stream buffers, while the parameter NSTREVENTS controlled the number of stream events. Under IRIX 5.x and IRIX 6.0.x, these parameters are now dynamically allocated.
System Table Entries for Device Status
The callout table is used by device drivers to provide a mechanism that prevents the system from hanging when a device does not respond. In IRIX 4.0.x, the NCALL parameter defined the size of this static table. Under IRIX 5.x and IRIX 6.0.x, the definition of this table has been made more flexible by defining several parameters:
ncallout the number of entries available at boot time callout_himark defines the maximum number of entries in the callout table reserve_ncallout the number of entries that are specifically reserved for kernel interrupt routinesIf a large number of device drivers are in use on the system, these parameters may need to be modified. For information on changing these parameters, refer to appendix A, "IRIX Kernel Tuning Parameters" in the IRIX Admin: System Configuration and Operation Manual.
Parameters that Affect Files
Under IRIX 4.0.x, the parameters NFILE and NINODE defined how many files and inodes could be open on the system at a given time. In IRIX 5.x and IRIX 6.0.x, these limits are now dynamically allocated.
Under IRIX 4.0.x, the parameter NOFILES defined the maximum number of files that a single process could have open at one time. In IRIX 5.x and IRIX 6.0.x, this parameter has been replaced with two parameters; rlimit_nofile_cur and rlimit_nofile_max. Rlimit_nofile_cur is the maximum number of file descriptors available to a process, and can be modified at process run time. Rlimit_nofile_max is the maximum value of rlimit_nofile_cur, and cannot be modified by the process. System administrators sometimes change these parameters to limit the number of files that can be opened by users. For more information refer to the manual page for setrlimit(2) and the discussion of the limit command in csh(1).
Parameters that Affect User Processes
The NPROC parameter specifies the number of entries in the system process table. In IRIX 5.x and IRIX 6.0.x, nproc is automatically configured based on the amount of memory available in the system, but can be modified to meet local requirements.
The parameter MAXUP is similar to NPROC, but defines the number of processes allowed per user login. Other than adding some safety checks against NPROC, no changes were made to maxup in IRIX 5.x and IRIX 6.0.x.
Under IRIX 4.0.x, the parameter SYSNCARGS controlled the number of characters on a command line. If this limit is exceeded, for example when commands containing wildcards expand to a large list of files, the system will respond with the error <command>: arg list too long.
In IRIX 5.x and IRIX 6.0.x, SYSNCARGS has been renamed to ncargs. Before changing this parameter, system administrators should consider using find(1) and xargs(1) to process large lists of files. Also note that some shells have internal limits for wildcard expansion. For example, the C shell limits the number of characters to a relatively small number regardless of the value of ncargs. Other shells, such as the Bourne shell abide by the limitation set by ncargs. Setting this parameter too large may cause some programs to function incorrectly.
Using Systune to Tune the Kernel
The systune(1M) command is used to change as well as display the current values of the kernel parameters. Some of the parameters can be dynamically changed; that is, the change is effective immediately without rebooting. Other parameters are static, and their new value will not be effective until the system is rebooted with the new kernel. Running systune with no options will display a list of all parameters as well as indicating which parameters are static or dynamic. When a static parameter is modified, a new kernel is generated (by default named /unix.install). Issuing the reboot(1M) command moves /unix.install to /unix and then restarts the system running the new kernel.
Because systune and reboot do not save a copy of the existing kernel, it is important that the system administrator manually save a copy of the kernel. Refer to the section earlier in this article titled "Save a Copy of a Working Kernel" for details on making sure that the system administrator has the ability to recover if a problem occurs.
When systune is used, the modified information is stored in the file /var/sysgen/stune. The default values for the operating system are stored in files in the directory /var/sysgen/mtune. When possible, systune should be used to modify the kernel values. The individual files in the directory /var/sysgen/mtune should never need to be edited by hand.
There are some parameters, mostly networking and SCSI parameters that cannot be changed using systune. For these parameters, the individual files in the directory /var/sysgen/master.d should be edited by hand. Be sure to record the original, and modified values. One way of doing this is by making a copy of the line that will be changed, and making it a comment using standard C style comments.
A Tuning Scenario
There are many different reasons why the system administrator might want to tune an IRIX kernel. Kernel tuning is often done because an application vendor suggests it when the application is installed, or because the system administrator has received complaints from users. This section will describe a simple tuning example. Note that this example assumes that the system administrator has already saved a copy of the old kernel, and that a copy of the current kernel parameter settings has been made.
The Problem
A user calls the system administrator complaining that they received the error message fork failed: too many processes.
The Solution
The system administrator looks up the error message in Appendix B of the IRIX Admin: System Configuration and Operation Manual and sees that there are two related parameters; nproc and maxup. In talking with other system users, the system administrator determines that this problem is not specific to any one user, and appears to be a system wide problem. Since nproc is the system wide parameter, the system administrator uses the systune command to determine the current value of nproc.
# systune | grep nproc nproc = 300 (0x12c)Based on the fact that in addition to the current users, the system administrator knows that new users will be added to the system soon, the decision is made to change the value of nproc to 500.
Sample Systune Session
% /bin/su - root
# systune -i
systune-> numproc group: numproc (statically changeable) ngroups_max = 16 (0x10) ncsize = 1200 (0x4b0) ncallout = 250 (0xfa) callout_himark = 532 (0x214) ndquot = 1200 (0x4b0) nproc = 300 (0x12c)
systune-> nproc 500 nproc = 300 (0x12c) Do you really want to change nproc to 500 (0x1f4)? (y/n) y In order for the change in parameter nproc to become effective, reboot the system systune-> quit
If the file /unix.install exists when systune is run, the following might be observed:
# systune -i Updates will be made to running system and /unix.install systune-> nproc nproc = 300 (0x12c) [dev/kmem] 500 (0x1f4) [/unix.install] systune-> quitIn this case, if the system administrator changes some of the previously changed parameters, the changes from the last session will prevail.
This would be an appropriate time to save the output from systune again for possible future reference. Make sure that the file name is different from the one used previously.
# systune | pr -h "systune output" | \ tee /systune.after | lpThe date on the output should be enough to differentiate this output from others. This will give the administrator a list to compare against if there are problems in the future.
Recovering From Unusable Kernels
If the newly tuned and generated kernel fails to boot, the system administrator will need to recover either using a previously saved kernel, or a kernel will have to be generated using the previously saved systune information.
Recovering from the Command Monitor
If the newly tuned and generated kernel fails to boot, the system administrator can choose Command Monitor from the System Maintenance menu, and boot the previously saved kernel.
If the saved kernel is in the root filesystem, use the command:
>> boot unix.saveIf the saved kernel is on an alternate filesystem, use the command:
>> boot dksc(0,2,7)unix.saveThis will boot the kernel saved earlier in this article. Note that the above command tells the Command Monitor to load the file unix.save from SCSI controller 0, disk 2, partition 7. This is the partition that was mounted as /disk2 in the earlier example. For more information on disk partition naming, refer to the manual page for dks(7M).
Note that the saved kernel was not referred to as /disk2/unix.save because the directory /disk2 is a mount point that only exists on the root filesystem while IRIX is running.
Recovering from Miniroot
If a backup kernel was not saved prior to tuning the kernel, or if for some reason the backup kernel would not boot (or was not accessible from Command Monitor), the system can be recovered by booting miniroot. This procedure requires that the system administrator have the correct version of IRIX on CD-ROM for the system, or that there is an bootable software distribution directory on the local network that has the correct version of IRIX for this system.
The chapters titled "Starting an Installation", and "Using the [Inst] Main Menu" and "Using [Inst] Supplementary Menus" in the IRIX Admin: Software Installation and Licensing, Manual contain detailed instructions on booting miniroot.
When miniroot has been loaded, the inst(1M) tool will be executed, and the inst menu will be displayed. It is strongly recommended that the system administrator select the commands admin and then shroot. Running any other commands, such as attempting to install system software, may obscure the existing problem.
Copying a Backup Kernel
If a backup kernel was saved on a filesystem that is not accessible from Command Monitor, it is possible to manually mount the appropriate filesystem and copy the saved kernel to the root filesystem.
# mount /dev/dsk/dks1d3s7 /disk1 # cp /disk1/unix.save /unix # sync # exitThe exit command will return control back to the inst menu, where the return and quit commands will exit inst and allow the system administrator to reboot the system.
Restoring a Kernel From Tape
If a backup kernel was made on local tape, it is also possible to extract the saved kernel from a tar, bru, or cpio archive from the local tape drive.
Creating a Kernel
If a backup kernel was not saved on either disk or tape, it is possible to generate a new kernel in miniroot. However, because the tools available in miniroot are only a small subset of the normally available commands, this can be tedious and time-consuming.
The system administrator will need to examine the output of the systune command that was printed before the kernel was modified. Because the systune command is not available in miniroot, the system administrator will have to use vi(1) to manually edit the file /var/sysgen/stune and restore the kernel parameters to their former values.
If root and /usr are two separate filesystems on the system, it may be necessary to manually mount the /usr filesystem.
# mount -c /dev/usr /usr # rehashUse the following command to edit the file. Depending on the CPU and graphics type, vi may come up normally (full screen mode), or it may come up in open mode.
# vi /var/sysgen/stuneFor more information on using vi from the textport, refer to the article titled "Using the Vi Editor From the Textport" in the March/April 1994 issue of Pipeline (available on-line with InSight if Support Advantage is loaded).
The file /var/sysgen/stune contains a line for each param- eter that has been modified. The system administrator can delete or comment out these lines to return the kernel to the factory defaults, or the value can be changed to the previous (working) value.
For example, to comment out a line, place an asterisk (*) at the beginning of a line:
* nproc = 500Once the appropriate changes have been made to /var/sysgen/stune, write the file, exit vi and use the autoconfig(1M) command to create a new kernel (/unix.install).
# autoconfig -fOnce the new kernel is built, type exit to return control back to the inst menu, where the return and quit commands will exit inst and allow the system administrator to reboot the system.
Continuing the Process
Once the system is successfully booted with the previous parameters, the system administrator must re-examine values that made the kernel unbootable, or unusable.
The references listed at the end of this article may help to determine why a particular parameter, or set of parameters caused a problem.
If tuning the kernel was the result of recommendations made for an application, the vendor of that application should be contacted for assistance.
Although tedious, the system administrator may wish to change the suspect parameters one at a time to determine which parameter is causing the problem. Alternately, increasing the value of the parameters in smaller increments may also help.
Finally, if the problem cannot be resolved, contact your system service provider.
Bibliography
For more information on kernel tuning, the Command Monitor and booting from miniroot, please refer to the following documentation.
IRIX Admin: System Configuration and Operation, particularly chapter 11 titled "System Performance Tuning", and the appendix A "IRIX Kernel Tunable Parameters", available on-line with InSight.
IRIX Admin: Software Installation and Licensing, particularly chapter 3, "Starting an Installation", and chapters 4 and 5, "Using the [Inst] Main Menu" and "Using [Inst] Supplementary Menus", available on-line with InSight.
IRIX on-line manual pages for: systune(1M), lboot(1M), inst(1M), autoconfig(1M), chkconfig(1M), and swap(1M).
In addition, the following books may be of assistance. The presence or absence of any particular resource should not be construed as a comment on its usefulness.
System Performance Tuning, Mike Loukides, O'Reilly & Associates, Inc., ISBN 0-937175-60-9.
The Design of the UNIX Operating System, Mauriceu Bach, Prentice Hall, ISBN 0-13-201799-7.
The Magic Garden Explained, Berny Goodheart and James Cox, Prentice Hall, ISBN 0-13-098138-
Unix System V Performance Management, Phyllis Eve Bregman and Sally A. Browning, UNIX Press ISBN 0- 13-016429-1.